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We perform spin-resolved and spin-integrated angle- resolved photoemission spectroscopy mea- 
surements on a series of compositions in the BiTl(Si_<5Se5)2 system, focusing on ^-values in the 
vicinity of the critical point for the topological phase transition (the band inversion composition). 
We observe quasi two dimensional (2D) states on the outer boundary of the bulk electronic bands 
in the trivial side (non-inverted regime) of the transition. Systematic spin-sensitive measurements 
reveal that the observed 2D states are spin- momentum locked, whose spin texture resembles the 
helical spin texture on the surface of a topological insulator. These anomalous states are observed 
to be only prominent near the critical point, thus are possibly related to strong precursor states of 
topological phase transition near the relaxed surface. 

PACS numbers: 



Three dimensional topological insulator is a novel 
phase of matter that has attracted much interests in 
the condensed matter physics community [l|-[3|. It is 
demonstrated that a topological phase transition (TPT) 
from a trivial insulator to a topological insulator occurs 
by replacing sulfur with selenium in the BiTl(Si_5Se5)2 
system N,j5|. Using photoemission probes, previous 
works |5| measured the electronic structure of the 
BiTl(Si_<5Se(5)2 system with relatively large 5 step {5 
step ~ 0.2), from which they observed that the band 
gap closes and reopen again (the so-called band inver- 
sion process) upon increasing 5 and therefore built up 
the frame of the TPT in the BiTl(Si_5Se5)2 system. 

Although the frame has been built, the systematic 
and detailed properties of the band inversion process 
and the TPT in the BiTl(Si_5Se5)2 system remain com- 
pletely unknown both theoretically and experimentally. 
For example, one aspect that is of particular interest is 
the precise chemical composition of the quantum criti- 
cal point {5c where bulk band gap becomes zero) of the 
phase transition. The system at 5c has been theoretically 
understood to have a 3+1 dimensional massless Dirac 
fermion [6], which is predicted to be multiply connected 
to many other undiscovered novel topological phases such 
as a topological Weyl semimetal |3: ls|- icici^ it is 
even experimentally unknown whether the TPT in the 
BiTl(Si_(5Se5)2 system is a sharp transition as expected 
in the simplest theoretical picture [2] , or the transition is 
intrinsically broadened so that no precise critical point 5c 
can be determined. An ideal physical property, in order 
to investigate such physical process, is the spin struc- 
ture across the TPT. In the sharp transition case, the 



spin structure is expected to show a sudden change from 
doubly spin degenerate (topologically trivial regime) to 
spin polarized (topological insulator regime). However, 
if the transition in BiTl(Si_5Se5)2 is indeed broadened, 
then the change of the spin channel across the TPT is 
completely unknown. In fact, it is possible that the spin 
transition is also softened in certain ways. For example 
the electronic structure even on the topologically triv- 
ial side can pick up spin polarization prior to the phase 
transition. 

In this Letter, we make the experimental efforts to de- 
termine these interesting physical properties along the 
TPT. Using angle-resolved photoemission spectroscopies 
(ARPES) and spin-resolved ARPES, we present sys- 
tematic electronic and spin structure measurements on 
BiTl(Si_5Se5)2 with 5 = 0.0, 0.4, 0.45, 0.5, 0.525, 0.55, 1.0 
respectively. Single crystals of BiTl(Si_5Se5)2 with var- 
ious 5 used for this measurements were grown using 
the Bridgman method, which is described elsewhere Q. 
High resolution (10-25 meV) ARPES measurements of 
the low-energy electronic structure were performed at 
the Synchrotron Radiation Center (SRC), Stanford Syn- 
chrotron Radiation LightSource (SSRL) and Advanced 
Light Source (ALS). Spin-resolved ARPES [10] measure- 
ments were performed at the SIS beamline of the Swiss 
Light Source, Switzerland using the COPHEE spectrom- 
eter with two 40 kV classical Mott detectors. Samples 
were cleaved in situ and measured at 10-80 K in a vac- 
uum better than 1 x 10~^^ torr. To calculate the kz 
of the photon energy dependence measurements, we use 
the relation kz = \^2m{Ekin cos^ + Vb) with inner 
potential Fo~12 eV. The theoretical band calculations 
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FIG. 1: Observation of quasi 
two dimensional states on the 
trivial side of the topolog- 
ical phase transition. (a)- 

(c) ARPES k-E maps of 
BiTl(Si-<5Se<5)2 for (a) triv- 
ial insulator, S = 0.0, (b) 
concentrations near the band 
inversion, S — 0.45, 0.50, 
0.525 and 0.55, and (c) topo- 
logical insulator, 5 = 1.0. 
ARPES spectra are measured 
at the f point along the f- 
K direction. White dots are 
guides to the eye for loca- 
tions of the conduction and 
valence bands. Spectrum 
for 6 = 0.45 is highlighted. 

(d) Energy distribution curves 
(EDCs) for 5 = 0.0, 0.525, 
and 1.0. The EDO through 
r point is highlighted. (e) 
A schematic diagram of the 
three dimensional Brillouin 
zone of BiTl(Si_5Se5)2 and 
the two dimensional Brillouin 
zone for the projected (111) 
surface, (f) Photon energy de- 
pendence spectra for 5 = 0.45. 
(g) kz dispersion maps for 5 — 
0.525 and 0.4. 



were performed with the LAPW method using WIEN2K 
package [11] within the framework of the density func- 
tional theory (DFT). The generalized gradient approxi- 
mation (GGA) was used to model exchange-correlation 
effects. For semi-infinite surface system, calculations are 
done based on a Green's function with implementation 
of the experimentally-based k • p model in Refs. |l2|, Il3 
to reveal the surface spectral weight and spin texture of 
BiTl(Si_^Se^)2. 

We start by showing electronic structure (spin- 
integrated) of the BiTl(Si_(5Se(5)2 system at different S. 
As shown in Fig. 1(a) and (c), the two end compounds 
{S = 0.0 and 1.0) clearly show the absence and the pres- 
ence of Dirac surface states connecting the bulk con- 
duction and valence bands, proving the trivial insulator 
{S = 0.0) and the topological insulator {5 = 1.0) phase re- 
spectively . The trivial insulator state has been shown 
to extend from 6 = 0.0 to 0.4, whereas the topological 
state is observed from S = 1.0 to 0.6 (see Ref. Q and 
the supplementary information). Now we focus on fine 
tuning of compositions in between 0.4 and 0.6, as shown 
in Fig. 1(b). A small (but still clearly observable) band 
gap of about 30 meV is observed onS = 0.45 in Fig. 1(b), 
indicating that the system still belongs to the trivial in- 
sulator regime. When increasing S to 0.50 and 0.525, the 
bulk conduction and valence bands are observed to fur- 
ther approach each other, and the gap is not observable 



in the dispersion maps [Fig. 1(b)]. The overall measured 
electronic structure at (5 = 0.50 or 0.525 in Fig. 1(b) 
shows linear (Dirac-like) dispersion. At 5 = 0.60 (shown 
in supplementary), we observe that the bulk conduction 
and valence bands are separated again by an observable 
band gap with the surface states connecting the band 
gap, demonstrating that the system belongs to the topo- 
logical insulator regime. The topological nature of the 
system at (5 = 0.50 or 0.525 (whether trivial or nontrivial) 
can not be unambiguously determined by the dispersion 
measurements in Fig. 1(b). The linear (Dirac-like) band 
can be the two dimensional topological surface states or 
the three dimensional bulk Dirac states [6[. 

Now we measure the electronic dispersion as a func- 
tion of incident photon energy. By varying the photon 
energy, one can effectively probe the electronic structure 
at different out-of-plane momentum kz values in a three 
dimensional Brillouin zone [Fig. 1(e)]. Fig. 1(g) shows 
the photon energy (kz) measurements at (5 = 0.525 by 
Fermi surface mapping in k\\ ys kz momentum space. The 
straight Fermi lines in parallel with kz direction show the 
absence of observable kz dispersion (for systematic mea- 
surements, see supplementary information), which seems 
to suggest that the Dirac bands at (5 = 0.525 are topo- 
logical surface states. However, far from the end of the 
story, we have also carried out photon energy dependence 
measurements at S = 0.45, where there is a clear band 
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FIG. 2: Observation of spin-momentum locked states on the 
trivial side of the topological phase transition, (a) ARPES k- 
E map of BiTl(Si-5Se5)2 for 5 = 0.40. Dotted line shows the 
binding energy where the spin-resolved measurements are per- 
formed [(d)- (e)]. (b) Momentum distribution curves (MDCs) 
of the spectrum in (a). Highlighted MDC curve is chosen 
for spin-resolved measurement, (c) Fermi surface mapping 
for 5 — 0.40. The two spin-resolved momentum distribution 
curves (SR-MDC) measurements are along V-M and V-K di- 
rections respectively. Yellow arrows represent the measured 
spin polarization vector around the Fermi surface, (d) Up- 
per panel (left/right): measured SR-MDC along V-M /V-K 
direction. Lower panel (left/right): measured in plane spin 
polarization along V-M /V-K direction, (e) same as (d) but 
for out-of-plane spin component. 



gap. As shown in Fig. 1(f) and (g), even for 5 = 0.45, 
no clear kz dispersion is observed, which indicates the 
quasi two dimensional nature of the observed bands even 
in the trivial insulator regime [5 = 0.45 or 0.40). Thus 
at (5 = 0.45 or 0.40, we observe a two dimensional state 
that marks the outer boundary of the bulk bands. The 
"true" bulk bands deep inside the bulk crystals is, how- 
ever, not clearly resolved in our measurements. This is 
because ARPES is a surface-sensitive probe (penetration 
depth within 10 A, see supplementary information). In 
narrow band gap semiconductor systems, the surface is 
typically subject to surface relaxation and band bend- 
ing effects [3, [lEI , which trap the bulk electronic states 
near the surface, and cause additional two dimensional 
iH, US- 



electronic states pJl, |17| . These effects make the "true" 
bulk bands deep inside the bulk crystal not resolved by 
ARPES. The fact that the observed electronic states at 
S = 0.45 or 0.40 are dominated by the two dimensional 
states at the outer boundary of the bulk bands must hin- 
der us from precise pinpointing the composition of the 
critical point Such observation implies that on the 
relaxed surface of BiTl(Si_(5Se(5)2 system, the evolution 
of the electronic structure across the TPT is far differ- 
ent than the simplest theoretical picture where the elec- 
tronic structure is expected to go from 3+1 dimensional 
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FIG. 3: Evolution of spin- momentum locked states with bind- 
ing energy and doping. (a),(e),(i) ARPES k-E maps with 
dotted line indicating the energy levels of spin-resolved MDC 
measurements. Doping levels are marked at the top of each 
map. (b),(f),(j) MDCs with highlighted curves chosen for 
spin-resolved measurement. (c),(g),(k) spin-resolved MDC 
spectra. (d),(h),(l) corresponding spin polarization measure- 
ments. 



massive Dirac bands (non-inverted trivial regime, fully 
gapped), to 3+1 dimensional massless Dirac bands (crit- 
ical point, gapless), and then to 3+1 dimensional massive 
Dirac bands (inverted nontrivial regime, topological sur- 
face states connecting the bulk gap) [6]. 

Now we perform spin-resolved measurements on the 
newly observed two dimensional states on the trivial side, 
in order to test if such states are spin polarized or not. We 
start from 5 = 0.40 [Fig. 2(a)] and focus on the vicinity 
of the Fermi level {Eb = —0.02eV). The momentum dis- 
tribution curves (MDCs) for the spectrum in Fig. 2(a) 
are shown in Fig. 2(b), where the highlighted curve is 
chosen for spin-resolved MDC measurement. Left pan- 
els of Fig. 2(d) show the in-plane spin-resolved momen- 
tum distribution curve (SR-MDC) spectra as well as the 
measured in-plane spin polarization along V-M direction. 
Clear in-plane spin polarization is observed on the two di- 
mensional states from Fig. 2(d). Furthermore, the mea- 
sured spin polarization in Fig. 2(d) (left) shows that 
the spin texture is arranged in the way that spins have 
opposite directions on the opposite sides of the Fermi 
surface. In order to map out the in-plane spin texture 
around the Fermi surface [Fig. 2(c)], SR-MDC measure- 
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FIG. 4: Theoretical calculation of spin-resolved spectral weight on the surface of BiTl(Si-(5Se5)2. (a) The energy levels of the 
bulk conduction and valence bands on the two ends {6 = 0.0 and 1.0) are connected by straight lines to show the general trend 
of the band inversion. The compositions chosen for spin-resolved measurements in Fig. 2 and 3 and calculations in Fig. 4 are 
marked by red and blue arrows respectively, (b) Theoretical simulations of the bulk and surface dispersions on a semi-infinite 
system of BiTl(Si-(5Se5)2 in the trivial phase prior to the quantum phase transition, (c) Same as (b) but for the topological 
insulator phase. Colors represent the spin polarization calculated by taking the difference between the up and down spin and 
dividing them by their sum [(/up — /down)/(/up + /down)]. Here 'up' spin is along the positive /c^-direction and vice versa, (c) 
Energy bands in presence of an external potential at the surface. Color displays their spectral weight. Magenta lines represent 
the surface Rashba cones now shifted inside the energy gap under the influence of the external potential. 



ment is performed along another momentum direction 
T-K. As shown in Fig. 2(e) right panels, the spin- 
resolved measurement reveal the same spin-momentum 
locking configuration along T-K direction. Correspond- 
ing out-of-plane spin polarization along f-M and T-K is 
shown in Fig. 2(e). No significant out-of-plane spin po- 
larization [Fig. 2(e)] is observed within our experimental 
resolution. The spin texture configuration obtained from 
these spin-resolved measurements in Fig. 2(d) and (e) is 
schematically shown by the arrows in Fig. 2(c). Surpris- 
ingly, our spin-resolved measurements reveal that these 
two dimensional states are not only spin polarized, but 
also their spin texture resembles the helical spin texture 
on the topological surface states as observed in Bi2Se3 
[l^ and BiTl(SoSei)2 {S = 1.0) [| from the Fermi sur- 
face point of view. 

After showing the spin configuration of these two di- 
mensional states at S = 0.40, we present systematic spin- 
resolved studies to show how the spin texture evolves 
as a function of binding energy Eb and composition S. 
We begin with 5 = 0.40 and show spin-resolved mea- 
surements at different binding energies. As shown in 
Fig. 3(a)-(d), the spin- momentum locking behavior is 
observed at all binding energies from near the Fermi level 
Eb = —0.02 eV to energy near the conduction band min- 
imum Eb = —0.32 eV. While the magnitude of the spin 
polarization on the Fermi level is found to be around 0.3, 
the spin polarization magnitude is found to decrease to 
almost zero when approaching small momenta near the 
Kramers' point f (the conduction band minimum). Now 
we turn to the gapless case with the system at S = 0.50. 
Again, spin-resolved measurements [Fig. 3(e)-(h)] reveal 
the same spin texture configuration, which resembles the 
helical spin texture of the topological surface states on 
the topologically nontrivial side. The magnitude of the 



spin polarization is around 0.5 on the Fermi energy, and 
does not show obvious reduction of magnitude when go- 
ing to small momentum near the Kramers' point T (spin 
polarization ~ 0.45 at the Dirac point, Eb = —0.32 eV). 
Finally, we turn to the composition far in the topologi- 
cally trivial side {S = 0.0). Our spin-resolved measure- 
ments [Fig. 3(i)-(l)] show only very weak polarizations 
(~ 0.05). The magnitude of the spin polarization is too 
weak (comparable to the instrumental resolution) to fig- 
ure out the spin texture configuration around the Fermi 
surface at 5 = 0.0 (We noticed an independent work also 
reports the near absence of spin polarization at J = 0.0 
in the BiTl(Si_5Se5)2 system [18].) 

In order to cross-check the results obtained from spin- 
resolved measurements, we have also created a Green's 
function implementation of the experimentally-based k-p 
model in Refs. lH, [l^ to simulate the bulk and surface 
dispersions of a semi-infinite crystal of BiTl(Si_5Se5)2. 
The spin-resolved surface dispersions are shown in the 
colorplots of Fig. 4(b) and (c) for trivial and non-trivial 
phases, respectively, the maximum intensity being the 
up spin and the minimum being the down spin along the 
^/-direction. The black dashed lines are the bulk bands 
obtained from a slab calculation. The white dashed lines 
represent the edge of the bulk bands in Fig. 4(b) and the 
surface states in Fig. 4(c). To determine the nature of the 
electronic states near the surface on the trivial side under 
band bending, we add an extra potential at the surface of 
the semi-infinite system. The resulting band structure is 
shown in Fig. 4(d) for 5 = 0.40. Now we clearly see that a 
pair of spin-splitting states (magenta dashed lines) cross 
at the F-point. These states are Rashba-like only close 
to the f-point, but evolve close to each other and even- 
tually become spin-degenerate at large momenta, unlike 
pure Rashba states. The calculated spectral weight of 
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5 = 0.40 in Fig. 4(d) is found to be dominated by the 
outer cone and the spectral weight of the inner cone is 
strongly suppressed possibly due to scattering with other 
bulk bands [19], in consistency with our ARPES measure- 
ments. The calculated spin polarization of the outer cone 
follows the left-handed helical configuration as shown in 
Fig. 4(b), also in consistency with our spin-resolved mea- 
surements in Fig. 2. 

We have shown the experimental observation and sup- 
porting calculation of a spin-momentum locked two di- 
mensional states on the trivial side of the TPT in the 
BiTl(Si_5Se(5)2 system. Calculation implies that such 
states are spin-splitted trapped bulk states (with inner 
cone strongly suppressed) cause by surface relaxation due 
to surface terminated electrical potential. In general, ad- 
ditional trapped bulk states due to band bending are not 



rarely observed, e.g. on Bi2Se3 surfaces [16|, ll7|. How- 
ever, two novel properties of the two dimensional states 
observed here have to be noticed: (i) The measured spec- 
tral weight of the two dimensional states arranges in such 
an exotic way (inner circle completely disappears) that 
the experimentally measured spin texture of Fermi sur- 
face (e.g. S = 0.4 spin data in Fig. 2) resembles the 
hehcal spin texture of the topological surface states, (ii) 
The spin texture on the trivial side is only prominent near 
the band inversion composition width (e.g. 6 = 0.4 — 0.5) 
but found to be significantly reduced for system far away 
to the trivial end with large bulk gap {S = 0.0). This 
strongly indicates that these two dimensional states on 
the trivial side are correlated with some form of prox- 
imity effect from the topological insulator regime across 
the phase transition. Thus the newly observed spin- 
momentum locked two dimensional states may serve as 
a strong precursor which show traces of the topological 
insulator state even prior to the occurrence of the TPT. 



In general, these experimental observations provide crit- 
ical knowledge to form a mature theory of topological 
phase transition from a trivial insulator to a topological 
insulator, which is lacking to the date. 
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